Devices and methods for programmable manipulation of pipettes

ABSTRACT

The present invention is directed generally to devices and methods for manipulating laboratory pipettes in a programmable manner. The present invention is directed to an apparatus and methods for allowing a user to instruct the device to perform a specific process; identifying the type, location and identity of the consumables to be used; manipulating a plurality of pipettes for performing the liquid handling; monitoring the process during and after its execution; generating a detailed report for the plurality of actions. Other aspects of this invention include optimization of the liquid dispensing performances of a pipette; monitoring and controlling individual actions by means of vision; virtualization of the protocol definition by means of a reality augmented software interface; integration of the system in a conventional laboratory environment workflow.

CROSS-REFERENCE TO RELATED APPLICATION DATA

This application is a continuation application and claims priority toand benefit of co-pending U.S. application Ser. No. 15/791,902, filed onOct. 24, 2017, which is a continuation application and claims priorityto and benefit of U.S. application Ser. No. 13/881,965, filed on Jul. 8,2013, and issued as U.S. Pat. No. 9,821,306 on Nov. 21, 2017, which is anational stage entry of International Patent Application No.PCT/IB2011/003037, filed on Nov. 22, 2011, which in turn claims thebenefit of U.S. Provisional Patent Application Ser. No. 61/416,546,filed on Nov. 23, 2010, the contents of each of which is incorporatedherein by reference in its entirety.

FIELD OF THE INVENTION

The present invention relates to the field of automation of chemical,biological, and biochemical processes or reactions. More specifically,it discloses devices and methods for manipulating pipettes in aprogrammable manner.

BACKGROUND OF THE INVENTION

Traditionally and historically, liquid handling constitutes afundamental building block of most biochemical, chemical and biologicaltests performed across multiple industries.

Liquid handling is essentially defined as the operations of putting onesample in contact with another one, sometimes in a repetitive way, beingable to quantify the amount of at least one of the two samples to beused. Despite the fact that a narrow definition of liquid strictlyindicates materials in the liquid form, hereafter we refer to liquidhandling the generic operation of handling materials in the solid (forexample powders), liquid or gaseous form- or in any mixture of thesestates (for example, heterogeneous samples containing solid and liquidsmixed together like cell cultures and emulsions or gases and liquidsmixed together like gels).

In the liquid handling arena, most solutions can be characterized bydifferent degrees of performances, where the performances are definedaccording different aspects which are of interest to the user, andconstitute a reason for utility: for example, flexibility, ease of use,throughput, reproducibility, traceability, and cost-effectiveness.Flexibility is defined as the capability of dealing with heterogeneousprocesses, over a wide range of volumes and for differentcharacteristics of the liquids, but also in respect to other propertiesand requirements. Ease of use is defined as the quality of requiringminimal training for its adoption, and a faster and intuitivetranslation of the user intent into the proper and desired operation. Inparticular, the translation of the user intent to perform a desiredoperation—without requiring a direct involvement during its execution—isalso referred to as programmability. Throughput is defined as the amountof independent, partially dependent or dependent processes that can beperformed within a suitable unit of time. Reproducibility is defined asminimal variations between different implementations of the sameprotocol for any reason. Reproducibility can be evaluated for protocolsperformed simultaneously or at a different moments by the same operatoror device, but it can also include variations introduced by differentoperators or different devices—in particular when evaluated with respectto the target performances as defined by the user, also referred to asprecision. For example, lack of precision in a biological process can begenerated by a slow clock used for the timing of the liquid handlingsteps—or by an incorrect calibration of the volumetric scale of theliquid handling device. Traceability is defined as the property ofkeeping record, for a-posteriori analysis and verification, of theactual process that has been implemented, including unpredictable eventsduring the protocol execution like possible faults or mistakes. Costeffectiveness is defined as the weighted sum of the cost components inthe acquisition of a liquid handling apparatus, user training, cost ofconsumables, cost of maintenance, cost of operations, cost of repair andcost of dismissal at the end of its lifetime.

Liquid handling today is performed either manually by human operators,or by means of automatic devices of various types.

In the most conventional laboratory environment, liquid handling isperformed by means of tools—defined as pipettes—allowing for aquantitative estimation of the sample being transported. In the case ofliquids, a common practice is to estimate the amount of sample by meansof its volume. Therefore, manual liquid handling is typically performedby means of volumetric adjustable pipettes capable of transportingliquid from one recipient to another in a known amount pre-defined by anoperator. Hereafter we define as pipette the liquid handling toolavailable and initially foreseen for the procedures of manual liquidhandling, or at least partially conceived for this application or simplyinspired to the tool used for this purpose. It should also be mentionedthat two types of pipettes are commercially available: electronicpipettes and mechanical pipettes. While electronic pipettes present someadvantages in terms of calibration and ergonomics, mechanical pipettesstill represent a large fraction of the market, being economical,performing, robust, cheaper and simple to operate. Above all, they'vebecome an industry standard tool responding to very precise norms, forexample ISO 8655 normatives. The difference in ergonomics is mainlyrelated to the force to be applied by the operator thumb (defined alsoas thumb action) on the pipette itself, for example for the purpose ofliquid aspiration, dispensing, mixing, and tip ejection. The overall setof procedures involving a pipette is hereafter referred to asmanipulation of the pipette.

In most cases, for the purpose of avoiding contamination, pipettes aretypically interfaced to the sample by means of tips, which areconsumables meant to avoid a direct contact of the pipette itself withthe liquid—that otherwise will unavoidably transport undesired moleculesto undesired places. The use of tips has become a standard practice inindustrial and research environments, with multiple types available andchosen by customers according to their maximum volume, presence offilters, surface absorption properties of molecules, materials, brandsand ultimately cost. Pipette tips can be considered specific pipetteaccessories or in alternative as part of a larger class of laboratorydevices defined as consumables, that include among others microplates,tubes, Eppendorf tubes, microtubes, vacutainers, filters, containers,capsules, vials and bottles typically used in the field of liquidhandling and biological or chemical reactions.

In recent years, the pharmaceutical, biotechnology, chemical, healthcareand related industries have increasingly adopted automated solutions forperforming various reactions and analyses. The benefits of theseautomatic devices include reproducibility, speed, capacity andultimately cost reductions at high throughput, enabling some users toperform a large number of reactions with limited human intervention,typically performing multiple reactions in parallel.

Automatic devices are usually associated to laboratories which requirelarge production capacity—since their size, cost and complexity ofoperations induce users adopting them when a significant number ofprocesses to be performed. However, sometimes automatic devices are alsoused in low and medium throughput environments, when the features ofreproducibility and traceability are strictly required-like in thesector of healthcare and diagnostics.

Examples of applications in the sector of healthcare consists in theprocessing of heterogeneous biofluids, defined as biological or chemicalfluids which present different components which are visually selectableat the macroscopic level. A known example consists of processingseparated blood, for example following fractionation, with the purposeof separating buffy coat from erythrocytes and plasma (or serum).Extraction of the buffy coat from the tube by manual pipetting is a veryunreliable, imprecise, difficult and time consuming operation.Therefore, blood banks employ dedicated automated systems of largecomplexity, like the one described by Quillan et al. (InternationalJournal of Epidemiology 2008; 37:i51-i55) which are addressing the needof precise and reproducible operations at high throughput. However, alsosmaller clinical environments, like hospitals and analysis laboratories,dealing with a smaller number of patient samples would profit of thesame advantages of reproducibility at a more limited throughput.

Cost of automatic devices is often linked to their mechanicalcomplexity: precise and reproducible movements over a large area requireprecision mechanics, including undeformable metallic frames determininga significant weight, ultimately making these systems not transportableand expensive to manufacture. Weight and dimensions has also asignificant impact on the cost of operations, since maintenance,repairs, training and upgrades have to be performed by specializedpersonnel on-site. And heavy systems imply stronger motors and higherelectrical current absorption, making their design more complex andexpensive to produce. Not to speak about portability of the devices andan easy integration into an existing laboratory.

Among others, a crucial requirement of a liquid handling process is itsactual reproducibility with respect to state-of-art validated protocols.Since most of the assay development is performed by means of manualliquid handling, it is obvious that results emerging from manual liquidhandling often constitute the reference for a given liquid handlingsystem. However, it is well known to those skilled in the art (forexample, Pandya et al.—Journal of Pharmaceutical and Biomedical Analysis53, 2010, pg. 623-630) that manual liquid handling misses in particulartraceability, precision and reproducibility. This is partially takencare by tools calibration and performances, since above all it isconsequence of the human nature and the propagation of instructionsbetween humans, training included. In addition, the low acquisition costof manual liquid handling tools should not hide the significant cost ofoperations generated by the necessity of having human operators. This isparticularly true since it also emerged that repetitive operationsinvolving pipettes introduce a significant strain on the musco-skeletalsystem, with possible consequence of work-related diseases. So, thepotential productivity of one operator has to be limited to minimize therisk of occurrence of different pathologies, like cumulative traumadisorders (CTDs) and repetitive strains injuries (RSis). Obviously, itwould be desirable to remove these risks completely from theprofessional environment—however the straight replacement of humans withautomatic liquid handling systems clashes against the need offlexibility, which is required in various activities, but also collideswith economic considerations due to the significant initial cost to beundertaken for the adoption and operation of automated infrastructure.In summary, there is the current evidence of a gap between manual liquidhandling operations and automatic liquid handling systems—whichultimately address in different ways liquid handling targets but do notoverlap in utility. The present inventions address this gap, providing auseful tool to research environments and industry.

Another crucial requirement of a liquid handling system consists in itstransportability, and a small space usage in a laboratory.Transportability enables a lower final cost to the user, avoidingon-site installation of the system and on-site support and maintenance.A system with a small footprint and light weight allows its installationin a conventional laboratory environment without the need of specificinfrastructure, and better integration into the existing laboratoryworkflow. A light system additionally absorbs less electrical current,enabling the possibility of battery or solar power in those areas whereelectrical supply is limited.

As pipettes, including state-of-art design solutions for the purpose ofmanual liquid handling, a summary of some of the prior art includes:

-   -   Gilson et al. (U.S. Pat. No. 3,827,305) teach a hand-help        pipette with adjustable volume mechanism;    -   Magnussen et al. (U.S. Pat. No. 4,905,526) teach an electrically        assisted pipette;    -   Scordato et al. (U.S. Pat. No. 4,821,586) teach an example of        computer controlled pipette;    -   Gilson et al. (U.S. Pat. No. 6,158,292) teach a tip ejection        system for a liquid handling pipette;    -   Cronenberg et al. (U.S. Pat. No. 6,977,062) teach an automatic        tip removal system including tip identification methods.

As automatic liquid handling systems, their engineering solutions andtheir conceptual design, a summary of some of the prior art is asfollows:

-   -   Gilman et al. (US 2003/0225477) disclose a modular equipment        apparatus and methods for handling labware    -   Pfost et al. (U.S. Pat. No. 5,104,621) disclose an automated        multi-purpose analytical chemistry processing center and        laboratory workstation.    -   Shumate et al. (U.S. Pat. No. 6,372,185) disclose a liquid        chemical distribution method and apparatus    -   Bjornson et al. (US2006/0127281) disclose a pipetting apparatus        with integrated liquid level and/or gas bubble detection.    -   Kowalski et al. (U.S. Pat. No. 5,139,744) disclose an automated        laboratory workstation having module identification means.

As other solutions, integrating automation into dedicated systems at lowthroughput, or describing dedicated systems to specific applications,the prior art includes:

-   -   Zucchelli et al. (U.S. Pat. No. 7,152,616) teach devices and        methods for programmable microscale manipulation of fluids;    -   Blanton et al. (U.S. Pat. No. 7,601,300) teach a compact        integrated system for processing test samples at low throughput        in a diagnostics environment.    -   Clark et al. (U.S. Pat. No. 5,482,861) teach an automated        continuous and random access analytical system;    -   Wegrzyn et al. (US2004/0241872) teach an optical detection        liquid handling robot system;    -   Ruddock et al. (U.S. Pat. No. 7,105,129) teach a liquid handling        robot for well plates using a powered anvil.

One drawback of prior art, in general, has been the difficulty toreconcile flexibility, in the form of fully programmable andconfigurable devices, with simplicity, in the form of low costmanufacturing and low cost operation, and reproducibility,characteristic of automated liquid handling systems.

The present invention meets the need for a flexible, reproducible,traceable, solution to perform liquid handling, at the same timeimproving the advantages of manual operations and introducing thebenefits of automation at lower cost.

SUMMARY OF THE INVENTION

The present invention is directed towards an apparatus and methods formanipulating pipettes in a programmable manner: we define the systemsand the devices exploiting those methods as liquid handling androids orsimply androids.

Accordingly, in one aspect of the present invention, a plurality ofpipettes is operated by an apparatus comprising a plurality of pipettes,at least one arm manipulating at least one pipette among the pluralityof pipettes, and one software interface allowing to define the liquidhandling protocol to be executed and governing the arm behaviour.

In another aspect of the present invention, it is disclosed a method forperforming liquid handling by means of a manual pipette, which isoperated automatically by means of a mechanical arm for the achievementof grabbing the suitable tip, setting the correct dispensing volume,aspirating a desired amount of liquid, dispensing a desired amount ofliquid, ejecting the tip.

In yet another aspect of the present invention, a camera is used for thepurpose of liquid handling by means of imaging a deck area from aplurality of angles and positions, simultaneously recognizing, measuringand localizing the consumables by means of their shape, dimensions,color, height, barcode, distinctive features.

In another aspect of the present invention, the camera is integrated inthe liquid handling apparatus and moving altogether with the armcontrolling the pipette movements, enabling the use vision to identifythe consumables and to exploit position information from the images toknow precisely the relative position of the pipette against theconsumable location.

In yet another aspect of the present invention, a device for processingbiological or chemical fluids, comprising a deck area comprising aplurality of consumables in given locations, where the locations areassembled in a flexible and ordered configuration.

In another aspect of the present invention, a method for volumetriccalibration of a pipette in a liquid handling android is achieved bydispensing a plurality of pre-set amounts of samples into at least onecontainer, by evaluating the actual amount of samples being dispensed,and by incorporating into the software interface the notion ofcalibration without modifications to the pipette.

In another aspect of the present invention, a method for improving thevolumetric reproducibility of a pipette in a liquid handling android isachieved by controlling the speed of the thumb action, modulated as afunction of the volume, of the position of the pipette piston and of thecategory of liquid being used.

In another aspect of the present invention, a method for improving thevolumetric reproducibility of a liquid handling android is achieved byincluding at least one sensor measuring humidity or temperature orpressure and refining the pipette calibration on the basis of the sensorinformation.

In another aspect of the present invention, a method for manipulating apipette in a liquid handling android is achieved by means of measuring,preferentially a-priori but also in real-time, the thumb actuationpressure as a function of the thumb position, and afterwards controllingthe thumb action only on the basis of the thumb position and speed.

In yet another aspect of the present invention, a method formanipulating a pipette comprises measuring the pressure of the thumbaction as a function of the position of the thumb, and operating thethumb based on its position only.

In yet another aspect of the present invention, an apparatus forprocessing of biological or chemical fluids, the apparatus comprising adeck to host consumables wherein the deck is of a foldable type or of aself-assembling type.

In still yet another aspect of the present invention, a method forprocessing biological or chemical fluids, where a camera allows imaginga pipette tip, the same tip being partially transparent to light,wherein the camera can visualize the liquid inside the tip, and theimage grabbed by the camera allows assessing the liquid volume containedin the tip, for the purpose of verification, volume determination,tracing and quality control.

In yet another aspect of the present invention, a method for processingheterogeneous biofluids like separated blood or separated milk orcell-containing fluids or beads-loaded liquids or suspensions oremulsions, where a mechanical arm allows manipulating a pipette, acamera allows imaging a pipette tip, a camera allows imaging thebiofluid, wherein the relative position of the tip with respect to thevarious biofluid components is extracted from the image and used inorder to control the aspiration and dispensing of a pipette in a certainlocation.

In still yet another aspect of the present invention, a method forprocessing biological or chemical fluids in a liquid handling android,comprising the simultaneous imaging of a pipette tip with respect to aconsumable by means of a camera, and using the information from theimage in order to determine the relative position in space of the tipwith respect to the consumable in order to manipulate the pipette.

In yet another aspect of the present invention, a method for determiningthe liquid level in a container comprising the imaging of an objectoutside the liquid, and comparing the images of the same object whilemoving towards the liquid surface, wherein the change in the objectimages procured by the contact of the liquid with the object allowsdetermining the location of the liquid level with respect to the object.

In yet another aspect of the present invention, a method for determininginformation about tips contained in a tip rack, comprising imaging thetip rack and identification of one or a plurality of tags within therack, where the tags provide information about the number, location ortype of tips within the rack.

These and other advantages, objects and features of the invention willbe apparent through the detailed description of the embodiments and thedrawings attached hereto. It is also to be understood that both theforegoing general description and the following detailed description areexemplary and not restrictive of the scope of the invention.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a three-dimensional drawing illustrating a liquid handlingandroid;

FIG. 2 illustrates the details of setting the desired volume of apipette by means of vision-based feedback;

FIG. 3 illustrates the method of ejecting a tip by means of a fixedfixture and movement of the pipette itself;

FIG. 4 illustrates the details of vision-based tracking of the liquidvolume being used;

FIG. 5 illustrates the detail of vision-based relative positioning ofthe pipette tip with respect to other consumables;

FIG. 6 is a three-dimensional drawing illustrating a second embodimentof a liquid handling android;

FIG. 7 is a three-dimensional drawing describing the details of a handwhich is capable to grab a pipette, actuate aspiration and dispensing,and eject a tip;

FIG. 8 illustrates the structure and composition of a domino deck basedon domino blocks;

FIG. 9 illustrates one embodiment for the use of computer vision for thepurpose of three-dimensional localization of the arm position;

FIG. 10 illustrates the capability of three dimensional localization ofthe arm position and application of a method for improving the intrinsicspatial resolution of the system; and

FIG. 11 illustrates one embodiment for the use of computer vision withthe purpose of localizing the available tips in a tip rack.

DETAILED DESCRIPTION OF THE INVENTION

The present invention relates to the manipulation of pipettes, as wellas a number of its applications. For the purpose of illustration, thedrawings as well as the description will generally refer to theapparatus addressing this solution as a liquid handling android.However, the means disclosed in this invention are equally applicable tomore general embodiments in the field of liquid handling.

General Description of a Liquid Handling Android

The global structure of a liquid handling android comprises fewelements, all of which have a given functional role in the architecture.In essence, a liquid handling android operates above a certain deck,that could include or not the android base itself. The deck could eitherbe a physical part, soft or rigid, either a virtual region withoutdelimitations—for example belonging to a laboratory bench. The deckcould also be the physical assembly of smaller units, called blocks,that combine together in order to form a larger operating surface. Aliquid handling android body—also referred to as base—provides thephysical support to the arm, and possibly may comprise additionalhardware like power chord connector, general switch, illumination,twister, settings camera, arm fixation, USB hub, tip waste tray, pipetterack, lifting handle. Most importantly, its purpose is providing acertain stable anchor to the arm movement. The arm constitutes the mainelectromechanical element: it generates movement of the hand in space,mainly moving over a two dimensional surface but also capable of liftingand descending the pipette in order to perform the desired pipetteaction. The arm is attached to the body and could either comprise ahand, or be connected to a hand. The hand constitutes the part of thebody coming in contact with the pipette, and with the optional abilityof grabbing and depositing pipettes onto the pipette rack. Additionally,the hand may contain a hand camera, the functionality of manipulatingthe pipette knob for the purpose of aspiration and dispensing, thefunctionality of tip ejection and the functionality of actuating thepipette for the purpose of setting a desired volume. The system iscomplemented by a software interface, whose purpose comprisescontrolling the movements of the arm, the actions of the hand,communicating with the cameras and processing the images, and above allmanaging all the interaction with the user for programmability purposesand also for reporting purposes.

A possible liquid handling android can be made as described in FIG. 1.The body 101 could be an injection moulded polymeric structure, eithermonolithic or in various parts, including active components (likeelectronics and cameras) and passive components like a weight ballast(solid or liquid-filled), preferably positioned in the lower part 104.In some embodiments, the body could include a foot (not shown) meant toprovide additional stability. In other embodiments the body could bepositioned onto a laboratory bench but, by hosting batteries andinterfaces, could also be designed to be used in other environments,like in the field or in portable implementations. In the figure, thebody hosts a ballast 104, a receptacle for a removable tip tray 103, abody camera in location 102 with the purpose of volumetric setting andpossibly with the purpose of deck area monitoring and inspection forintrusion detection, a plurality of pipette slots 121, in the form ofreceptacles or hanging fixtures or magnetic holders or similar designedto host pipettes like the one indicated by 105. The body can include alifting handle like the one described in 108, and mechanical elementslike the one indicated in 109 making the interaction with the pipetteeasier, for example by allowing an easy access the ejection button forthe tip. The body can include a twister 122, defined as the actuatorcapable of setting the pipette volume. Typically, this operation isperformed by twisting the knob of the pipette, but it could also beachieved by electronic means for electronic pipettes—for example remoteBluetooth communication or physical electrical links. It should be notedthat additional electronics accessories could improve the advantages ofthe system: for example, a temperature or pressure or humidity sensor,possibly connected to a USB hub and read directly from the softwareinterface, could allow improving the calibration of the pipettes byintegrating and correcting for this information.

The deck area 106 defines the operating surface of the liquid handlingandroid, being larger, smaller or equal than the operating range of thearm. The deck area could have a circular shape, a rectangular shape orsimilar. Preferably the deck has a shape making intuitive to the userthe correct orientation. The deck could be a virtual region, for exampledelimited by simple illumination, but also a soft pad (for example, asilicon pad that can be easily rolled above itself to reduce its sizeand recover a flat conformal shape when positioned onto a bench), or arigid metallic or polymeric plate, including wood or compositematerials. It is important to emphasize the possible advantages ofvirtual or foldable decks, since portability of a liquid handlingandroid constitutes a main advantage for service and support operations,making the shipment of the android more effective cost-wise. Inaddition, a foldable or virtual deck allows saving space when theandroid is not in use. The deck could contain a plurality of locationsproviding specific information, either to the user either to the systemitself. For example, labels, warnings, instructions, precautions, anddisclaimers addressed to the user, but also localization marks,barcodes, coded symbols, tags, fiducial spots, to improve the spacelocalization of the pipette and the consumables by means of the cameras.A plurality of types of consumables, for example the microplatesindicated as 107, can be positioned onto the deck, either in a freeformat configuration, either in fixed or almost fixed formatconfiguration. A fixed format configuration implies to preciselylocalize the consumable in a given position, without leaving anarbitrary choice for its orientation, while an almost fixed formatconfiguration indicates an approximate region for the consumable, butleaving the option of rotations and displacements in proximity of thenominal position for the same. Fixed format configurations may profitfrom slots, rails or similar solutions. In all configurations, thepresence of serigraphic or printed graphics can facilitate the user jobof positioning a plate, but also simplifying the function of consumablelocalization by the cameras and providing a sense of order to the userperception, making the repetition of the same protocol an easier task.Optionally, the printed graphics and information could be performed indifferent colors, making the camera more selective to identify a part ofthe information hereby present.

The arm, in this case defined as the structure between element 110 andelement 113, comprises a plurality of actuators or solutions with asimilar functionality (for example, a cable driven system where themotors are actually localized outside the arm, or a pneumatic systemusing cylinders as actuators). In the present embodiment, actuators arechosen from the category of servo motors integrating gear reduction andangular feedback, allowing setting the actuator to a given angle betweenits body and the output axis. In a single unit, for example unit 110,the provision of power and serial communication link (for example basedon the RS232, RS485 or USB standards) allows to input and outputdifferent information: examples of input are the desired position, thevelocity profile for a movement, the maximum torque, the angularacceptance window; examples of output are the current position, thecurrent velocity, the unit temperature, the unit status, and possiblefaults. The motion of the arm occurs mainly in the horizontal plane,being typical biochemical operations performed on a planar andhorizontal bench with consumables which have a marginally differentheight. However, the insertion of tips and the aspiration and dispensingof liquids, for example, also require vertical movements. In thisspecific embodiment, the arm operates mostly in the horizontal plane andit has a more limited excursion in the vertical plane. One way toachieve the required displacement, for example, would be relying on twoangular actuators setting the position in the horizontal plane and avertical linear actuator. In alternative, the weight and complexity ofthe linear actuator could suggest its replacement by two angularmovements, for example the angular actuators 112 and 113, allowingmoving the pipette up and down by conserving its orientation in spacethrough simultaneous movement. This feature can be important inconsideration of the fact that the pipette verticality constitutes animportant requirement for better volumetric performances of pipettes.For other reasons, it could be preferable to increase the number ofangular actuators for a movement in the horizontal plane. For example,in some embodiments it could be desirable to define the orientation ofthe vertical pipette with respect to azimuthal rotations: thisautomatically implies at least three actuators for horizontal movements.The presence of obstacles or fixed structure could also require a largernumber of actuators, for example four as depicted in FIG. 1. The choiceof the arm configuration could follow good engineering practice andcommon sense, in view of the application and of the angular actuatorsperformances.

The hand design could exploit concepts and components similar to thoseapplied to the arm. In the depicted embodiment, the hand starts fromactuator 114, which is actually the actuator taking care of the grabbingof the pipette. The grabber, not shown for clarity, can be a simple clawmechanism capable of exercising a pressure on the two sides of thepipette. It could also be a single claw mechanism, where the moving clawis opposite to a fixed claw which is conformal to the pipette. Claws canhave, in general, a conformal shape, a planar shape, or a limited numberof contact points with the pipette. Different design have differentadvantages: depending on the embodiment, the liquid handling androidcould be designed to deal with a single type of pipette, or with amultiplicity of models. It is obvious to those skilled in the art thatclaws have to be conceived accordingly, and their conception could bedifferent for different pipettes. The hand may further comprise a camera123, to be oriented and moved in different directions, independently ordependently together with the pipette, with the purpose of identifyingthe consumable and its position in space but also the position of thetip 120 or the pipette 119 once it has been grabbed from the body slot121. It is important to realize that it is challenging to image, with afixed camera, a typical deck surface characteristic of a biological orchemical test without going too far away from the deck. Therefore, thesuggested embodiment indicates a solution for the problem by imaging thedeck area by a series of pictures individually covering a part of theuseful surface. The image could be recomposed in a mosaic by suitablesoftware, allowing having a synoptic view of the deck space and theconsumable thereby contained. The composite imaging could also allow—bytilting or translation of the camera or of the hand—to have multipleimages of the same deck or part of it. This feature could be easilyexploited with the purpose of obtaining stereoscopic information inorder to reconstruct at least part of the three-dimensional information.This feature is particularly relevant in order to extract information onthe height of the consumable, possibly required for the correct settingof the pipette aspirating and dispensing position. Three-dimensionalinformation could also be achieved by means of using the focusinformation from the camera, provided that the camera has an adjustablefocus and the optical configuration has a limited depth of focus. Thismethod, would allow extracting depth information by simple scan of theobject itself, and analysis of the spatial contrast of the image. Acolor camera could also provide additional information, for exampleallowing identifying consumables and pipettes or other accessories basedon the color space distribution. The hand may include a thumb actuator115, whose purpose is to actuate the thumb 116 with functionalitysimilar to the human thumb in the manipulation of a pipette. The thumbmovement could be a simple partial rotation around the axis, but it isimportant to notice that improving the precision of the thumb action,for example in its speed, position, and pressure sensitivity withrespect to a human thumb, could introduce various improvements in thepipette manipulation: for example, improved ml×mg of liquids by rapidaspiration/dispensing sequences through the excursion of knob 117,improvements in the precision of dispensing by a reproducible positiondisplacement or velocity profile, and an improved detection of thepipette stop by pressure feedback mechanisms. Ultimately, the thumbaction could also depend on the liquid properties—making the pipetteworking in optimal conditions with viscous liquids or heterogeneoussamples. As another example, a fast and reproducible thumb action couldimprove the performances and the reliability for on-the-fly dispensingof liquids, defined as dispensing of liquids without physical contactwith the recipient-contained liquid. This possibility would enableperformances that are not possible to be achieved by manual pipettingoperations, with significant savings in time and in the use of tips. Acombination of a multiplicity of dispensing and aspirating methods,combined with the possibility of individually calibrate them forarbitrary liquids (as described in a following section) supports theevidence that a liquid handling android can outperform easily a manualoperator, both in capacity and quality.

A second embodiment of a liquid handling android is described in FIG. 6.A plastic enclose 601 constitutes and contains the main body, which isdesigned as a vertical structure mounted onto a baseplate 602. Thebaseplate 602 has the purpose of providing stability to the system, andto make the system independent from possible vibrations and oscillationsof the supporting bench—whether induced by the android itself or byexternal agents. The body 601 also include a rotating actuator 603 forthe execution of the volume setting procedure. The rotating actuator isassisted by a camera 604 that, by means of the internal illuminator 605,is capable if imaging the digital counter positioned onto the pipettes606. In this embodiment, the body 601 contains electronics andmechanics: in fact, the vertical movement of the arm is achieved by alinear actuator (not visible in the picture) that raises vertically theshoulder 607, allowing for the required vertical excursion of the arm.As a consequence of this, the arm functionality is limited to thedisplacement of the hand 608 in the horizontal plane, being the verticalmovements achieved inside the body 601. Differently from FIG. 1, the armtherefore contains only three servomotors 609 that allow for completecoverage of the intended area.

Details about the hand embodiment are shown in FIG. 7. Two servomotors701 and 702 assist the hand in manipulating the pipette, includinggrabbing, ejecting the tip, and actuating the pipette knob 705 foraspiration and dispensing of liquids. Servomotor 701 has the doublefunction of applying the required pressure on the pipette knob 705,including the monitoring of the pressure feedback and the monitoring ofthe knob position in order to determine the pipette stop. The doublefunctionality is achieved by means of cams, where cam 704 is alwaysmoving together with the servomotor 701 axis, while the cam 712 isactuated by the cam 704 only within a limited angular range. Thepressure of cam 704 onto cam 712 actuates the button 706 on the pipette,inducing the ejection of tip 709 from the pipette. Another cam isactuated by servomotor 702: cam 703 actuates a lever (not shown) thatslides on wedge 707, which in its turn pushes the clamp 708 against thepipette body and results in the pipette grabbing. A symmetric mechanismis present on the other side of the pipette, resulting in a symmetricclamping force aligning the axis of the pipette with the axis of thehand.

Importantly, the hand hosts a camera 711 and an associated light source710. The purpose of the light is to apply uniform and constantillumination in the field of view of camera 711, field of viewcomprising the bird flight view of the deck, the imaging of the tip 709and in this case also of the pipette end 713. Having these elementswithin the field of view, allows measuring the relative position ofthese objects within the camera image. In fact, the correction of theoptical distortion of the lens allows determining the radialline—passing through the objective of camera 711—along which an objectwithin the field of view lies. Therefore, its transversal position canbe reconstructed by estimating its vertical location. The verticallocation of an element, for example the tip end, can be estimated indifferent ways: by means of the lens focus, by contact of the sameobject against a reference of known vertical position (sensed throughthe pressure feedback of the vertical motion), by multiple displacedimages of an object which is not connected to the hand, by stereoscopicimaging of two cameras are mounted on the hand, by measurement of theapparent size of a 2-dimensional barcode of known dimensions, and othermethods.

Detailed Description of Volumetric Setting

A possible embodiment describing methods and devices for the definitionof the pre-set volume in an adjustable pipette is described in FIG. 2.In the picture, a camera 203 is located inside the body 201, bodyalready described in FIG. 1. The camera is positioned in such a way tobe able to image the pipette display 215 (not directly visible in thepicture being covered by the pipette body but indicated for example inlocation 313 of FIG. 3) indicating the dispensing/aspirating volume ofpipette 204. Obviously, the arm which is partially visible (actuators213 and 214) has been suitably designed in order to allow this positionto be reached. The camera could either image the display from the front,or from a certain angle in whatever direction and plane (for example,from the top or from the bottom, from the left or from the right). Thecamera could be assisted by artificial illumination, either from theenvironment or from sources contained in the liquid handling android,either from natural sources. It is useful to combine the displaymonitoring with the capability of adjusting the pipette volume setting.This is accomplished by the actuator 206 connected to the knob twister207. The actuator can be set either by its angular position, either byits angular velocity. The knob twister is an element, preferably ofelastic material, which has been designed in order to be able, by simplepressure of the knob against the twister, of applying a torque on theknob therefore allowing—as done for the majority of pipette types—toperform the required pipette adjustment. In some embodiments, thetwister could be a rubber based cylinder with a concave (truncated) conecarved into its body: the cone shape would allow to conformallyadjusting to different sizes of pipette knobs.

Detailed Description of Tip Ejection

A possible embodiment describing apparatus and methods for the action oftip ejection is shown in FIG. 3. Obviously, tip ejection in a liquidhandling android is complemented by tip insertion onto the pipette.However, in most of the present pipettes the tip insertion is simplyperformed by applying a certain pressure when the pipette body has beeninserted into the tip. Clearly, this operation is feasible in anembodiment as described in FIG. 1. Concerning the tip ejection, multiplesolutions could be exploited, including the direct action of theejection button by means of a dedicated actuator most probably locatedinto the hand of the liquid android. However, there is an economicalsolution which doesn't require an additional actuator, as shown in FIG.3 for the liquid handling android embodiment already described inFIG. 1. The arm allows localizing the pipette 303 in a configurationwhere the ejection button 305 of the same pipette is facing a fixedstructure 306, for example fixed with respect to the body structure 301.The actuation of the ejection button is achieved by a force generated bythe arm itself, for example by the action of the actuators 309 and 310in order to have the fixed structure 306 and the ejection button 305being pushed one against the other. This solution allows saving at leastone actuator and a certain complexity in the hand, resulting in alighter and more reliable solution. An appropriate choice of the shapeof the structure 306 allows also ejecting the tip in different spatialposition, something which is desirable to avoid the accumulation of tipsinto a limited area of the waste tray 103 shown in FIG. 1.

Detailed Description of Volumetric Monitoring

A possible embodiment of methods and devices achieving volumetricmonitoring and traceability of pipetting operations is shown in FIG. 4.The four images correspond to four different snapshots taken by acamera, which in the liquid handling android previously described couldeither be camera 123 or camera 102 of FIG. 1. For simplicity ofdescription, the image is taken from a position which is orthogonal tothe pipette axis: however, this is not strictly required and most anglesof view are possible. The image can visualize in part or in full thepipette body 402 and the tip 401. As it is visible in the leftmostimage, a reference image of an empty pipette constitutes the referenceand it could also be stored-temporarily or permanently. It is understoodthat the image could be taken in a reference position of the arm, soproviding a uniform and constant background information andillumination.

In the second picture from the left of FIG. 4, it is depicted a tipwhich has been loaded by a given amount of liquid, according to thevolumetric settings of the adjustable pipette. It is obvious to thoseskilled in the art that every set volume corresponds, for a given tip,to a given location of liquid meniscus 403. In this respect, therefore,the meniscus location constitutes an indicator that the pipette hasaspirated correctly the desired amount of liquid.

Conversely, the reference image constitutes the logical reference aftera dispensing operation, where the presence of droplets or liquidleft-overs can also be detected in a similar way. In the third picturefrom the left in FIG. 4, it is shown a pathological case where theaspiration is not occurred correctly. Visibly, a bubble of air 405 hasbeen introduced in the pipette, modifying the actual liquid volumecontained in the pipette with respect to the desired volume. Accordingto the origin of the bubble, the meniscus 404 could be at the correctposition (defined according to the considerations done for the secondpicture from the left of the same figure), therefore indicating that theactual liquid volume in the tip is lower than expected. The liquidmeniscus could also be at a higher level, indicating for example thatthe bubble has been formed after aspiration, or could even be lower thanexpected—suggesting a serious problem in the liquid collection. A simpleand practical case occurring in laboratory practice is shown in therightmost picture of FIG. 4: a lack of liquid in the container where thepipette has aspirated the liquid, or the incorrect position of the tipwith respect to the liquid level, has resulted in a partial aspirationof the liquid at the advantage of air contained in the pipette. Themeniscus 405 is most probably in the correct position; however a secondliquid-air interface is visible in location 407. All these undesirablebehaviours can be made available to the user, significantly improvingthe interpretation of the data generated by the assay. In all cases, theimage contains significant information that would be lost in manualoperations. This useful information could either be processed online, inorder to try recovering the process, either simply stored offline foroperator monitoring and quality control purposes. Overall, a similarimaging configuration could be used for controlling the position of atip in a consumable with respect to the liquid level. The imaging of theconsumable, and the identification of the liquid level, could allowdetermining the vertical distance between the liquid and the tip,allowing precise sipping or dispensing of liquids. Similarly, the sameprocedure could be applied to aspirate liquid in particular verticallocation of the liquid, for example in the case of separated blood andaspiration of buffy coat at the interface between plasma/serum anderythrocytes.

Detailed Description of Vision Assisted Tip Positioning

A possible embodiment describing methods and apparatus for achievingvision assisted positioning of a tip is shown in FIG. 5. The imagecorresponds to the image taken by a camera which is preferably connectedto the pipette hand, for example camera 123 described in FIG. 1. If thecamera is connected to the pipette hand, grabbing the pipette 119connected to tip 120 in FIG. 1 will result in a reproducible andconstant position of the pipette tip 504 visible in FIG. 5. Therefore,this information constitutes already an important control on the propergrabbing of the pipette from the hand. It is understood that differentpipettes and different tips could result in different images and shapes,so the tip imaging also represents a possible method for making surethat no misidentification has occurred. Additionally, the image maycontain—as in the case of FIG. 5—additional objects within the field ofview. It is well known in the art that any object could be either infocus, either out of focus, accordingly to the type of optics and sensorutilized, and obviously their distance from the camera. The armcapability is such that it is possible to operate the arm at a desiredheight, which of course means that the distance between the consumableand the tip will be set to a desired value. In this conditions, it ispossible to identify the lateral alignment of tip 504 with respect tothe desired well position 507 according to the following method: theaxis 504 of the tip 503, when prolonged, will identify the trajectorythat the tip will perform for a vertical movement (in the example thatthe tip is vertical, as it should typically be). However, a given andtypical distance of the tip with respect to the consumable will define asingle point in the image that the 15 tip will intersect when localizedat the same height of the identified well. Therefore, the relativehorizontal alignment of the tip can be achieved by imaging the same tipwithin the field of view, and applying an offset in the imaging plane:this point should be directly positioned onto the desired destination,by applying lateral movements of the arm without changing the distanceof the tip from the consumable. It should be remarked that this methodworks also in presence of optical distortions, that can be correctedeither in full by vision analysis method either by empirical alignment.

In another implementation, as visible for example in FIG. 7, the cameracan image the tip while the tip is approaching the liquid surface. Withrespect to an image where the tip is far away from the liquid, an imagewhere the tip is in contact with the liquid will change the image of thetip, and therefore such change can be used to identify the positionwhere the tip touches the liquid surface, for example with the purposeof aspirating or dispensing nearby the liquid surface.

The difference in the images can be enhanced by suitable illumination ofthe tip or of the liquid: as soon as they come in contact, therefraction index of the tip polymer and the refraction index of a liquidare similar, and therefore light will channel through the other mediumunder the guidance of internal reflection along the materials surface.The change in the illumination configuration can be easily identifiedand lead to the detection of the tip-liquid contact. Illuminationconditions particularly suited to the internal reflection exploitationcan be achieved by means of light emitting diodes or lasers, or underthe guidance of light guides, like for example optical fibres.

Detailed Description of the Domino Deck

A possible embodiment of a deck configuration is shown in FIG. 8.Differently from the deck described in FIG. 1, the consumables areorganized in a geometrical manner by means of holders called blocks,defined as reusable or non-reusable supports capable of holding one or aplurality of consumables. A feature of the blocks is the possibility ofassembling them into a larger structure called mosaic, which is a planarcomposition of blocks organized according to some pre-defined rules butwith a certain pre-defined flexibility. In FIG. 8, different types ofblocks are assembled together: for example, block 801 IS intended forthe collection of used tips, bock 802 is designed to contain and supportdifferent types of microtubes, block 803 is intended to hold and supporttip racks, block 804 serves the support of larger tubes like for example15 mL, 50 mL and blood tubes, and block 805 contains a microplate. Theseblocks are not exhaustively covering all possibilities. For example, ablock could be designed to host simultaneously pre-loaded reagents,specific consumables like tips, barcodes for processing information,tubes and empty consumables for allowing the users providing their ownsamples. In this last configuration, it is possible to conceive a dominoblock as a single unit that doesn't require external blocks forprocessing, making therefore the domino deck a collection of independentexperiments that do not depend from each other. Importantly, dominoblocks can be complemented by information by means of NFC, RFIDs, linearbarcodes, optical recognition marks and two-dimensional barcodes asindicated in 806. The purpose of providing additional informationreliefs the system in active and contact-less identification of theblocks, for example by means of the camera 711 described in FIG. 7.Other ways of extracting the domino block information is by means ofelectrical contacts positioned on their sides and coming into contactwith neighbouring blocks, and propagated to the other blocks by means ofan electrical network. One important feature of a domino deck consistsin the capability of adapting its configuration to the user needs, whilesimultaneously being able to organize and rule the assembly of theblocks. In fact, the domino block could present keys on the sides, forexample mechanical keys or magnetic keys, preventing the user fromassembling the domino block incorrectly, and also validating the choiceof a configuration by some forces keeping the assembly all together. Oneembodiment for a key is a mechanical configuration, similar to thoseimplemented in LEGO toys for the purpose of education and play. Anothermechanism consists in specific magnetic configuration: for example,along a side designed to be oriented in direction “down” the side couldhost a plurality of magnets presenting a suitable magneticconfiguration. Poles in the configuration SNS (South-North-South) couldbe matched to sides presenting NSN (North-South-North) as a consequenceof an attraction force, while sides NSN will be pushed away from a sideNSN (similarly to the repulsion force of a SNS side when pushed againsta SNS side). The advantage of a magnetic configuration consists in anattractive force validating an allowed configuration, while a repulsiveforce will prevent assembling blocks with the wrong orientation. Thesemagnetic forces could also improve the overall organization of thedomino deck by means of connecting to an external reference structure.For example, in FIG. 8 the block 807 is magnetically attached to thebase below the body of the android by means of a SNS magneticconfiguration facing a NSN magnetic configuration generated by magnetsembedded on side 809. Similarly, block 808 is magnetically connected tomagnets position on side 810 of the android base through an SNS magneticconfiguration facing a NSN magnetic configuration. In this example, whatprevent to rotate by 90 degrees the block is the different pitch betweenmagnets, shorter on side 810 with respect to side 809. For the samereason, the blocks in the domino deck cannot be rotated by 180 degreesor by 90 degrees.

One important advantage of a Domino deck consists in an optimal spaceoccupation of the laboratory bench, being external to the android body.In fact, the space occupied by the system is limited to the spacerequired by a given experiment, contrarily to the configuration of todayliquid handlers that occupy bench space irrespectively of the complexityof the experiment involved. Additionally, it allows minimizing theoccupied bench space when the system is not used, for example by storingthe domino blocks elsewhere or by assembling them in a vertical pileoccupying the footprint of a single domino block. In general, users canexploit different domino blocks according to their typical experiment,by varying the amount of blocks of the various types which are requiredand without using the blocks which are unnecessary.

Detailed Description of the Space Localization of the Arm

While multiple procedures and methods for positioning are known to thoseskilled in the art, including the use of precision mechanics andencoders and decoders of X-Y-Z Cartesian robots, we describe a methodwhich is particularly suited for the identification and localization ofconsumables by means of a simple camera mounted on the moving arm. Thecamera and arm geometry here described is the one shown in FIG. 6, thearm 609 holding a pipette 608 by means of the grabber 708 shown in FIG.7 together with camera 711 and related illumination 710. FIG. 9represent a possible image taken by camera 711 while moving above acertain block, to be accessed precisely for the purpose of pipetting inone given position (for example well 910). It should be noted it iscritical to extract the relative position of the pipette axis in threedimensions with respect to the desired pipetting location. Known thepipette tip length (for example by the pipette model or by othertechniques including the sensing of the pipette tip contact,stereoscopic 1 magmg, external measurement by means of camera 604 ofFIG. 6, and other methods), and given the fact the pipette tip could bevisible within the field of view of the camera (as possible in FIG. 7for pipette 709 by exploiting a suitable objective for camera 711), itis evident that the lateral position of the pipette tip end with respectto the camera axis can be computed in the space of the image sensorcoordinates (pixels) and converted in real space lateral displacementonce the conversion scale is known for the plane where the pipette tipend resides. The conversion scale can be achieved in multiple ways,including the use of a two-dimensional barcode of known dimensions inthe same plane. However, FIG. 9 shows that knowing the relative positionof the pipette tip end with respect to the camera is a partial solutionto the problem addressing the positioning of the pipette tip end into awell 910, since it is still required to move the camera axis (shown bythe hatched cross 901) at a given offset (in the real space) withrespect to the consumable 902. The following method shows a procedurewhich has the advantage of being rapid and robust, being capable ofcompensating any misalignments and locally adjusted for each individualblock or small area of the deck. In fact, block 911 is equipped withdifferent features. One feature is the presence of mirrors 903, 904,905, 906 positioned on planes which are at 45 degrees with respect tothe horizontal plane, and reflecting in the upward direction the imagefrom the side of the microplate. These mirrors allow the opticalinspection of any user-labelled barcode put on the vertical sides of themicroplates, that can be measured by the camera 711 easilyirrespectively of the sides where the barcode is applied, andpotentially detecting any microplate rotation if the user barcode shouldbe in a given side of the microplate. The same barcode identificationcapability can be exploited to detect other barcodes implemented intothe block 911 in positions 909 and 908 for example. It should beemphasized that the choice of two barcodes could be reduced to a singlebarcode and could be extended to a plurality of those, with the purposeof increasing the system robustness or the amount of information to beread by the camera. The two dimensional barcodes mounted in block 911are positioned at about the same height of the wells, or at a knownoffset in the vertical plane. The reading of a barcode, for example of aQR barcode, also provides to the user information about its apparentsize, which is the size measured by the camera in its space (typically,measured in pixels along the directions of the dimensions of thesensor). Having barcodes of known dimensions, or of dimensions which arereported into the content of the barcode itself, allows therefore todefine the spatial conversion scale to convert any distance measured bythe camera in the same plane into real dimensions. Alternatively, if thebarcode is of unknown dimensions, two barcodes at a known distance canserve the same purpose, for example by knowing the distance betweenbarcode 908 and barcode 909. Indeed, for the case of camera with unknownpixel shape the information about the barcode angle has to be used inorder to extract the suitable conversion scale (different in the twodirections of the image sensor). In summary, measuring the dimensionsand the angle of a single two-dimensional barcode allows for measuringdistances in the same plane of the barcode, or in its proximity.However, for a given camera and objective the conversion scale changeswith the distance according to simple projective rules, once the cameraimages are corrected for the objective distortions. So, a vertical scanperformed by moving vertically the camera at known steps (for example,knowing the gear factor and the steps of a motor moving the armvertically, allows to construct a curve that automatically provides theuser, by interpolation and extrapolation, the vertical distance for agiven camera and objective from the barcode itself. Ultimately, the samecurve can be used in a reverse manner to extract the actual distance ofthe camera from the barcode, and knowing the offset of the pipette tipend with respect to the camera: this inverse method allows solving theproblem of vertical positioning of the pipette end tip with respect towell 910.

Similarly, the lateral offset of the camera axis 910 with respect towell 910 can be computed by knowing the lateral offset of well 910 withrespect to a barcode 909 in the reference frame described by arrows 912and 907. This offset is specific to each module, and can be stored intoa suitable way externally or internally to the module (for example, bymeans of a database, inside the barcode data, or by an RFID or NFC tag).To achieve the target of relative positioning of the arm, it should benoted that the camera axis 901 is localized in the reference frame 912and 907 by the measurement of the barcode angle, its position in thesensor image and the previously described spatial conversion scale: thetransformation between the camera reference frame and the real spacereference frame of the block become uniquely identified by a singleimage. So, putting all elements together, the present method allowsprecise relative positioning of a pipette with respect to a location ina given consumable by means of a camera mounted on a robotic arm, usingthe information provided by a barcode.

In fact, the present method can be used also for precisely identify theparameters transforming the angles of the servomotors 609 of FIG. 6 intorelative coordinates within one block. This approach has the advantageof precisely refine the mechanical precision in presence of the armtwisting, bending, imperfections in the angular determination,inaccurate arm sizes and dimensions, assembly inaccuracies and ingeneral improving the reproducibility. In summary, the non-linear,non-invertible transformation of the servomotors angles into the cameraposition in the real space depends on a large number of externalparameters, but is a known analytical function following basictrigonometric rules. However, many of these parameters are more accuratewhen computed locally, for example the bending of the arm could vary asa function of the arm configuration (its extension, for example). Themethod here disclosed is based on multiple images of block 911—similarto the image of FIG. 9—where the images are displaced by a known, localangular amount for any of the motors, allows creating a dataset ofImages where the barcode position and angle is measured within thecamera Image. Using the arguments explained before for a single image,the distances between the theoretical positions of the camera in thebarcode reference frame 912 and 907 and the actual distances can beminimized by means of a least squares minimization algorithm, andtherefore an optimal local transformation can be created and usedafterwards. This procedure can be repeated rapidly over time, forexample triggered by a large discrepancy between theoretical positionsand actual positions during the arm operations, in order to maintain thesystem highly reproducible. In FIG. 11, an example of the residuals thatcan be obtained by changing individually the angular position of thethree servomotors 609 shown in FIG. 6, for a number of angular settings(each of the intersecting lines corresponding to the modification of theangle of one individual motor, as indicated in the plot labels). Thearrows in the plot indicate the residuals, defined as the deviation ofthe expected position vs. the actual position measured by the camera,after having applied the previously mentioned minimization procedure.The size of the arrows (magnified by a factor 5× in order to make themvisible in the plot) indicates the error in positioning of the system.The method here described allowed improving the spatial precision of thesystem by a factor 6×, taking average residuals from 6 mm (mainly givenby the precision of the mechanical system and of the electronics) toless than 1 mm.

Detailed Description of Tips Identification and Localization

A problem specific to liquid handling instrumentation is the need ofidentify, localize, count and dispose the liquid handling consumablecalled tip. There many different types of tips—and typical liquidhandling operations imply the disposal of the tip after each liquiddispensing step, to avoid further contamination. The consequence is acomplex logistics even for relatively simple protocols, both in manualoperations and liquid handling performed by automated systems. Inparticular, pipette tips in some disciplines have also strictrequirements in terms of sterilization and contamination beforeoperations actually have place: the consequence is that a typicallaboratory has a very complex tip management logistics, induced bymultiple tip types, compatibility of each tips for each equipment andmanufacturer, and of the formats and packaging associated to those.Essentially, all instrument manufacturers supply users with their owntip racks, tip rack being the name for a structure organizing tips in aregular array, and try to offer the widest choice possible in order toallow any operation on any instrument. Consequently, tips supply becomesan expensive activity both for users and instrument suppliers.

Hereby, we describe a novel solution allowing our androids to use anytip which is already being used in the laboratory. The solution istotally independent from the tip rack, e.g. the holder containing thetips. The solution allows also to identify uniquely the tips, and toknow which tips are usable in a rack without the requirement (demandedby most instruments) to start operations with unused and new tip racks.In this way, evident economy can be achieved by the customer,simultaneously obtaining the maximum flexibility in using high qualityconsumables on the android.

The solution consists in identifying and localizing tips by means oftop-view vision, for example the one achieved by means of camera 711 inpicture 7. Any tip rack can be positioned in a domino block like the oneshown in FIG. 11, the domino block essentially being a simple box(possibly with a surface with an anti-slip pad to avoid undesiredmovement over time of the tip rack itself) capable of hosting the vastmajority of tip racks commercially available. It is very common topurchase tip racks that organize the tip consumables in the samegeometrical configuration of microplate wells, e.g. a rectangular arrayof 12×8 tips which are spaced apart by 9 mm. Assuming thisconfiguration, in order to be able to use tips effectively we need todeal with various aspects: the identification of the type of tips, theidentification of the available tips, the determination of the height ofthe tip upper part that will come in contact with the pipette end. Evenif these operations could be performed by direct image processing, e.g.vision-based algorithms identifying shapes and structures, it would behard to be robust enough to be able to deal with hundreds of differentconfigurations and designs which are not known a-priori.

Our vision-based solution consists m inserting into the tip racks twobuttons 1101 and 1102. The buttons could be either inserted by the userbefore executing an experiment, but also before autoclaving the tips forfurther reuse, or at manufacturing. The two buttons could be made indifferent ways: as a simple cork to be inserted into a tip of thecorresponding type, or as a passive stub similar to the upper part of atip and having about the same external diameter. Buttons would require abarcode or similar optical mark at the top, the barcode being an easyand robust solution for identification and localization by thetop-vision camera mounted on the arm. The advantage in usingtwo-dimensional barcode consist in the fact that they will automaticallyprovide the precise vertical position of the tip for grabbing, and alsothe correct transversal scale for identifying the conversion scale inthe image allowing to reconstruct spatial dimensions. Spatialcoordinates are needed both for guiding the movement of the arm in orderto grab a tip, but also to compute and determine the number of availabletips, and their localization. In fact, barcodes 1101 and 1102 would beused to define the region of the tip rack where tips are present. In theexample from FIG. 11, all the 34 tips localized in the matrix defined bythe two buttons as corners would become a region from which the arm willpick the tips, region highlighted in the picture by means of the dashedrectangular perimeter 1103. It is evident by anybody skilled in the artthat the suitable choice of the corners would allow choosing the regionof the tip rack to use, and allow counting (by means of the known pitchamong tips) the number of tips available. Similarly, the content of thebarcode would provide to the system the information on the type of tipsbeing hosted in the specific rack. The method here described by means oftwo barcodes, can be easily extended to a plurality of barcodes anddifferent methods for indicating the usable sector of the rack for tipsextraction. This method therefore provides the way of localize, identifyand count tips in a substantially generic tip rack, and the sameprinciple could be used for the extraction of partial information—forexample in combination with tip recognition methods to discover possibleholes in the tips formatting (as an hypothesis, one tip being absent inlocation 1104).

It should be noted that the same method can be applied to differenttypes of consumables that imply picking operations: for example, needlesfor the purpose of liquid handling could be considered under the samemethodology, with equivalent advantages.

Detailed Description of the Software Interface

An important element of the liquid handling android is constituted bythe software interface, a generic name including the packagecommunicating with the camera, actuators and electronics, controllingand synchronizing their operations, processing the information to besent and collected, but in particular interacting with the user andexternal sources of information (websites and servers, for example). Theinteraction with the user consists both on the system programmabilityand the provision of feedback related to the liquid handling process,including its execution performances, faults, checkpoints. In onepossible embodiment, the cameras and the actuators of the liquidhandling android are controlled by means of USB, and a USB hub islocalized inside the body. In this embodiment, a single USB cable canconnect the personal computer or the tablet constituting the userinterface to the liquid handling android itself. In other embodiments, aWi-Fi connection could serve for the purpose avoiding the necessity of aphysical link. The controlling software could therefore exploit USBdrivers and software development kits provided with the individualcomponents with the purpose of minimizing the development, and similarlyintegrate existing packages for the vision processing and for theinverse transformation determining a set of actuators angles for a givenposition, in angle and space, of the pipette.

An important aspect of software is constituted by the user interface.The availability of cameras capable of capturing real images of theprocess suggests using an approach based on virtual reality, where theuser is provided with information—on the screen of the controllingsystem—which results partially from real images and partially fromsynthetic information. In this way, the adherence of the originalprotocol can be made in a more user-friendly way, improving theperformances of the operator and reducing possible faults to mm1mum.

The software interface could also interact with the user during theexecution of liquid handling steps. For example, a protocol couldrequire specific liquid handling steps—or operations likespectrophotometry, phase separation, microscope inspection orsimilar—which cannot be executed from the android itself. Therefore, thesoftware interface will trigger the user intervention (or in alternativesimply wait for it) for example by means of visual indicators, handwaiving, acoustic signals, emails, SMS or phone calls to the user.

The purpose of the software is not limited to the execution ofprotocols, but it could also be extended to other operations having, forexample, the purpose of improving the hardware performances. Forexample, it is well known in the art that accurate pipette performancesrequire frequent calibration of the same, being the performances relatedto environmental parameters and also to their use. A liquid handlingandroid could be controlled by software in such a way to execute pipettecalibration procedures for example repeating a sufficient number ofdispensing steps into a consumable, and monitoring (by weight,colorimetry, fluorescence or similar techniques) a physical parameterrepresentative of the dispensed volume. It should be noted that—in aliquid handling android—there is no strict need of physically adjust thepipette calibration scale, since the software could automatically definethe calibration table, and therefore the knowledge of the actual volumeto be set in order to achieve a desired volume.

Having now described a few embodiments of the invention, it should beapparent to those skilled in the art that the foregoing is merelyillustrative and not limiting, having been presented by way of exampleonly. Numerous modifications and other embodiments are within the scopeof ordinary skill in the art and are contemplated as falling within thescope of the invention as defined by the appended claims and equivalentsthereto. The contents of any references cited throughout thisapplication are hereby incorporated by reference. The appropriatecomponents, processes, and methods of those documents may be selectedfor the present invention and embodiments thereof.

What is claimed is:
 1. An apparatus for processing biological orchemical fluids, comprising a deck area comprising a plurality ofconsumables; a camera mounted on a moving arm, said moving arm adaptedto manipulate at least one pipette, wherein said camera is capable ofimaging from a plurality of locations or angles to acquire threedimensional information of the deck area; and a software interfaceinterfacing with said moving arm and performing an analysis of the threedimensional information of the deck area to recognize the consumablesand to manipulate and localize the moving arm with respect to theconsumables.
 2. The apparatus according to claim 1, wherein themechanical arm is adapted to manipulate the at least one pipette, themanipulation being assisted by the three dimensional information; andthe software interface governs the manipulation of the mechanical armwith respect to aspirating and dispensing based on the analysis of thethree dimensional information.
 3. The apparatus according to claim 1,wherein the software interface is adapted to define a liquid handlingprotocol to manipulate and localize the moving arm with respect to theconsumables according to the liquid handling protocol.
 4. The apparatusaccording to claim 1, wherein the camera is further adapted to image oneof the at least one pipette to determine if said pipette has aspirated adesired amount of liquid
 5. The apparatus according to claim 1, whereinthe camera is further adapted to image one of the at least one pipetteto determine a vertical distance between a liquid within said pipetteand a tip of said pipette for a precise dispensing of said liquid. 6.The apparatus according to claim 1, wherein focus information from thethree dimensional information is used to determine a height of at leastone of the consumables.
 7. The apparatus according to claim 1, whereinthe camera images the deck area from a plurality of angles andpositions, simultaneously recognizing, measuring and localizing theconsumables by a combination of their shape, dimensions, color, height,barcode, distinctive features.
 8. The apparatus according to claim 1,wherein one of the plurality of consumables is recognized by a tagpositioned onto a consumable holder.
 9. The apparatus according to claim1, wherein a relative position of the arm with respect to a consumableof the plurality of consumables is extracted by at least one propertyamong the three-dimensional information, orientation and size within animage.
 10. The apparatus according to claim 9, wherein a distance of thecamera from the consumables is determined from the three dimensionalinformation.
 11. The apparatus according to claim 9, wherein a lateralposition of the camera with respect to the consumable is measured froman apparent position of at least one tag within an image.
 12. Theapparatus according to claim 1, wherein the arm manipulation includesmovement of the at least one pipette, tip insertion and ejection,setting of the desired volumes, aspiration and dispensing of liquids.13. The apparatus according to claim 1, wherein the manipulation of theat least one pipette by sequences of aspiration and dispenses allows forliquid mixing.